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.Abstract 



Content analysis of critical events methodology is' demonstrated in an 
evaluation o:: the Educational Telecommunications for Alaska (ETA) Project.- 
This methodology was used to collect and interpret the data derived from ".he 
ETA staff personnel representing five components of the project. The data 
were analyzed using content analysis procedures . The results indicated 
that content analysis of critical events is a potentially useful and low 
cost method for collecting and analyzing easily available information and 
can be employed to provide project management with evaluation information 

.i. 

that may otherwise be overlooked. Suggestions for overcoming limitations 
of the methodology are included. ' g 
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Introduction 



The Educational Telecommunications for Alaska (ETA) Project is a 
multimillion dollar program emphasizing educational uses of telecommunications 
technology. The first year of operation witnessed two specific evaluation 
problems. Although these problems were specific to the context of the ETA 
Project, Connolly, Dale, Hart, : .nd Zelman (1975) have cited similar problems 
in other large-scale programs. 

As in all evaluations, the first task for the evaluation staff was to 
find a way to determine a method which could- obtain data that were readily . 
available about project progress and an appropriate methodology for • 
analyzing them. Although this task is present in all evaluations, it was 
of a larger magnitude in the ETA Project due to the vastness of Alaska. 
In a state where costs are 15 to 40 percent higher than in the lower 48 
states,. distances enormous, a..i the typical evaluation strategies 
questionable, collection of readily available data on large-scale program 
activities becomes formidable. For example, Alaska has four time zones, 
is 240Q miles from east to west and 1420. miles from north to south, and 
currently has approximately 450,000 residents scattered over., 586,412 square 
miles (Tacts about' Alaska, 19771. Furthermore, because of difficulties 
■in sending and receiving messages, exacerbated by harsh weather, data 
collection techniques such as mail surveys proved highly unreliable .. In 
1176, the Department of Education conducted a statewide, survey of school 
personnel and obtained a total response rate of only 36 percent 
(Planning and Evaluation Survey, 1976)., 
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The second task, also common to all evaluations, was to find a way 

to provide project management with formative (Scriven, 1969) information 

to aid in decision making while documenting project activities. Owens_, 

Haenn, and Pehrenbacher. (1976) have indicated that while emphasis has been 

placed upon providing information for decision making, much evaluation 

methodology has not kept pace with project management's needs. 

It was in response to these two tasks that content analysis of critical 

events was developed. Content analysis of critical events provides a 

framework to analyze information that is easily obtainable, low in cost, 

anci not overburdening to staff personnel. It also provides evaluation 

information to project management that might ordinarily be Overlooked during 

traditional evaluations and can be utilized for constructing evaluation 

questions that require future in-depth * study* — r 

Content Analysis and Critical Events 

Content analysis of critical events stems from 3erelson's (1954) early 

work on content analysis and Hecht's (1976) and Miackeracher , Davie, ..and 

Patterson's .(1976) works with critical events. Consent analysis has been 

employed primarily in the assessment of a wide variety of communication 

phenomena such as propaganda, trends, styles, changes in content and 

readability CKerlinger, 1973), although it has also been utilized' in 

program evaluation (Owens et al. , 1976) , 

Critical event analysis has drawn limited attention although it has 

been addressed in evaluation of telecommunications programs (Practical 

Concepts Incorporated, 1977) , While both of these methods have been . 

employed individually , a review of the literature did not indicate their 

* 9 
combined use in program evaluation, 
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Definition 

— i 

Berelson (1952), as cited by Festinger and Katz (1959, appeared to 
provide the most appropriate definition of content analysis for the present 
purpose: "Content analysis, is a research technique for the objective,, 
systematic, and quantitative description of any. symbolic behavior" (p. 424). 
Content analysis in this definition refers to a system of classification 
which allows for the objective, systematic, and quantitative .description. 

t 

of behaviors. It provides a framework for the researcher to: (1) identify 
primary characteristics of the content, (2) make inferences about the nature 
of the content, and (3) interpret the content so that it is meaningful. 

To define critical events, we modified a definition utilized by 
Practical Concepts Incorporated (1975) . jTcrdtical event in the ETA 
Project was defined as any significant program issue, action, or event 

in which a staff member participated during the past week. Employing 

this definition of critical events provided a model for the evaluators to 
systematically^collect, integrate, and. document significant actions o:c 
program members. The. collection of. information could come from direct 
observation, interviewing, or collection of documents. 

/ 

Procedure 

Kerlinger (1973) has specified three steps in content analysis. The 
first consists of defining the universe of content to be analyzed, the 
second involves categorizing it, and the third. involves quantification of 
the collected material. 

For the purpose of this study, the universe was defined as all 
replies to the question, "What significant program issues', actions, or 
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events have you participated in during the past week? " These responses 

\ 

were submitted by all of the five major project components via weekly 
status reports. The major project components, included: administration, 
instruction , management, the Department of Education (DOE), and the Regional 
Resource Centers (RRCs) • The administrative component consisted of members 
of the. 1 funding agency responsible for financial management but not part 
of the day-torday management decisions. The instructional component 
consisted of staff members actually implementing the day-to-day programs 
that were being developed. The management component included the project 
director and assistant director whose purpose was to make sure that the 
staff vas completing tasks on time. The DOE wr.s a separate component in 
charge ,of implementing the project on a statewide basis. The RRCs were 
the interface between- the DOE and local education districts. A checklist 
was constructed indicating whether the report was received by the evaluation 
-staff. All copies were filed for future analysis. The reporting procedure 
was initiated July 1, 1978 and an analysis was conducted after three 
months of operation. 

The categorization of the universe was developed by grouping into 
clusters key stimulus words derived from the responses from the five 
project components. This procedure produced four' mutually exclusive 
categories based upon the key stimulus words found in the responses. 
Fox this project, the procedure was referred to as a natural, clustering 
technique CLee, 1978).. A fifth category was then added to allow for 
classification of responses which did not fit into any of the final 
four categories. The categories and their corresponding definitions are: 



Content-. Analysis 

Implementation was. identified by key stimulus words such' as finished, 
finalized, delivered, implemented, installed, completed, etc. 
Process was identified by key stimulus words such as preparing, 
planning, investigating, exploring, meeting, projecting, reviewing, 
analysis, etc. 

Clarification was identified by key stimulus words or phrases such as 
change in policy, new position, change-in program direction, notification 
of new policy, etc. 

Problems were identified by key stimulus words or -phrases- such as 
had to respond to, mad, angry, criticised, annoyed, etc. 
Miscellaneous was included for any responses that could not be 
classified into any of the above categories. 

Analysis of the Data 
The unit of analysis chosen as most appropriate for evaluating the 



status Sports was themes. Kerlinger (19731 has indicated - that a theme 
is often a sentence, or proposition about something. In our case, a 
theme corresponded to one of the five categories cited previously. A 
response was recorded as falling , into. a 'particular category if it had 
characteristics sUnilar to that category definition or theme. 

To assess reliability of coding, Pi 00. was utilized (Scott a 
Wertheimer, 1962) . This procedure represents the extent to which two 
coders agree beyond the level that would be expected by chance. Responses 
from the status reports were coded by two independent judges based upon 
definitions of each category. . 
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Results 

Table 1 depicts the frequency of critical events for each category 
occurring in each project component. This table clearly indicates the 



Insert Table 1 about here 



majority of critical events reported for all project components occurred in 

% 

the two categories " Implementation " and "Process." Table 1 also indicates 
that the area with the least reported critical events was the "Problem" 
cateqory. These results were confirmed using follow up interviews. 

The computed reliabilities for the Administration, Instructional, 
Management, Department of Education, and Regional Resource Centers 
components were .88, .85, .76, and ' .66 respectively. The reliability 
for nominal-scale intercoder agreements among all components was .81. 
This was obtained by averaging all individual component reliabilities. 

Discussion 

Upon analysis of * the status reports, it was found that most activities 
which project participants reported could, be classified into one of the 
four maiA categories. These categories reflected the activities that 
participants were engaged in during data collection. The resulting 
* activities provided an indication of where most activity occurred and 
which components experienced problems. This analysis also produced o 
results that were used to alert project management to the different ways 
project staff was spending its time; These results proved particularly 
liseful when project management's expectations were not congruent with 
staff activities. 
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In addition to the analysis • conducted in this paper, a secondary 
analysis of any of the areas could also be provided. Fo4 example, any 
area that was particularly troublesome could be identified and -answers 
provided to questions such as, "Why does the problem exist?", "Who 
"caused it?, or "When did it start?" This information would b* derived 
from individual analysis of status reports and forwarded to project 
management. 

Although the authors feel that this methodology is useful for both 
formative and summative evaluation, two specific problems were identified.. 
The first problem was defining "critical events." Participants 'tended to 
view "critical" in a different way. What was critical to one participant 
was not necessarily critical to another. Also, what was not critical one 
day may have become critical one week later. It is suggested that the' 
definition problem be resolved by operationally defining critical events 
.more thoroughly and by conducting a preservice training session to explain, 
elaborate, and answer any questions that project participants have about 
the definition, 

A second problem existed with the category labeled "Process." It 
should be obvious from an examination o^f Table 1 that an extreme number 
of responses fell into thi,s category (as compared to other' categories) and 
that it did not adequately differentiate activities in this area. It is 
suggested that this problem be alleviated by constructing narrower, more 
discrete categories that adequately reflect each specific unit of a • 
theme. In this case, that could be" accomplished by further partitioning 
the category "process." 

Evidence of content analysis of critical events previously being 
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employed in program evaluation was quite scarce. It was developed as 
an adjunct method for facilitating data interpretation and. increasing 
formative evaluation information to project management. As a 'new research/ 
evaluation technique, it has accomplished its purposes, although further 
refinement will be required. It is hoped that this very .uccinct 
treatment of content analysis of critical events has indicated, the possible 
promise and problems -associated with its utilization and will stimulate 
its further development. Although this method has only been employed ^ 
briefly, the authors feel that it holds much promise' for future 
evaluation activities. 



) 



■A . 



ERJC ; 



11 



Table 1 

Total Frequency of Reported Critical Events 
by Project Component 



Zategory 



Admin. 



Project component 



Instruct. 



Manage . 



Implementation 

Process 

Clarification 

Problems 

Miscellaneous 



13 
65 
1 
0 
3 



9 
57 
1 
0 
3 



5 
42 
8 
3 

■ 1 
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